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(GSiC) 


Agenda 


ODS - Intros and back-office announcements 


DEEP — Update and Background 
DEEP — Demo to share Live Chat 
Break 

Q&A continued 


MPBSD - ServiceOntario: Enabling Chat Solutions Mid-beta DA-433 


Problem Statements 
Original chatbot 


+ How might we create an improved 
experience for the public seeking information 
on Driver and Vehicle Licensing Services 
(D&V) services that curates trusted answers 
and is available 24/7. 


Current Live Chat 


+ How might we improve the experience in the 
current D & V chatbot for the people who are 
looking to quickly self-service for 
information, or to get reassurance on what 
they find online about Driver and Vehicle 
services. 
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Digital Enterprise Engagement Practice (DEEP) (G S ° C) 
Government Services Integration Cluster l 


What has changed since we last connected? 


* New project name - MPBSD - ServiceOntario: Enabling Chat Solutions 

D&V Chatbot is now managed by ServiceOntario and DEEP and ITS provide support and 
guidance 

* ServiceOntario and DEEP are partners for Live Chat (Phase II), with SO leading the 
implementation and DEEP leading the technical and DFA components. 

* Transformation in GSIC and we are now the Digital Enterprise Engagement Practice (DEEP) - 
working beyond contact centres and focusing on new ways to engage the public 

New sponsor — ADM, Nelson Loureiro of Customer Care — with a strong focus to digital 


* Customer care is building their digital team and project team and lead the timing of Live Chat 
implementation 


Transformation and election with updated mandates 
* ODS get a welcome to our new Ministry of Public and Business Service Delivery 
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Digital Enterprise Engagement Practice (DEEP) (G S ° C) 
Government Services Integration Cluster l 
Background 


With the scope of Driver and Vehicle Services, Chatbot piloted from January 2021 until February 
of 2022 when ServiceOntario assumed operational management 


The purpose is to help Ontarians looking for Driver and Vehicle information by providing 
immediate access to answers they are looking for, as well the reduction in the volume of calls to 
the contact centre, increasing contact centre capacity for more complex calls and new business. 


It allows people to ask a quick question (non-personal), or select a button with a commonly asked 
question, to get a simple fast answer with latest information on driver and vehicle products and 
services. 


Live Chat was planned to follow the Chatbot, both research and the pilot confirm the desire for 
Live agent, and ODS/DEEP now requires Live Chat with Chatbot. 


The Live Chat implementation date will be determined by ServiceOntario who are in the planning 
stages right now 


4 MPBSD - ServiceOntario: Enabling Chat Solutions Mid-beta DA-433 GovTechON Ontario Y 


MPBSD 23-071 Record 04 ODS_DFA-AddQuestions- 2022-07-11 Final Page 5 of 37 


Digital Enterprise Engagement Practice (DEEP) (G S ° C) 
Government Services Integration Cluster l 
Approach 


* The focus of this current update is Live Chat technical and to operationalize the chatbot with 
ServiceOntario. 


Chatbot is now managed in the Change Management and Implementation team in the ADMO for 
Customer Care in ServiceOntario. 


> DEEP has prepared and tested the technology for Live Chat, integrating the current chatbot (IBM 
Watson) with queueing capabilities to get to a Live Agent. 


* Already have and use with other clients like ITS ServiceDesk with EVA (Chatbot & Live Chat)and Employment Ontario 
with Live Chat 


» The enterprise design standard for chat solution was used. 
* Developed with ODS and DEEP over the last few years and has involved many rounds of user research, and learning 
from the chatbot pilot 
> Our sponsor has committed resources — they are here today and will step in as the Live Chat PM lead 
the for operational and content development and implementation in partnership with GSIC-DEEP 
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Digital Enterprise Engagement Practice (DEEP) (G S ° C) 
Government Services Integration Cluster l 
Milestones 


January 21, 2021 — Launched Chatbot on one webpage F/E 


* April 08, 2021 — Fully launched on five webpages F/E 


December 2, 2021 — ServiceOntario PMO for Live Chat project 

January 2022 — Scope change Technology preparedness for Live Chat with SO resource constraints 
February 28, 2022 — ServiceOntario assumed ownership of the chatbot 

April 1, 2022 — ServiceOntario launched Two webpages - sticker refund F/E (approved) 

May 2022 — ODS led user research on the Live Chat technology 

May 2022 — Live Chat technical component up to system integration testing completed 


* June 2022 — Re-engaged ServiceOntario and Resources assigned for planning 
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Digital Enterprise Engagement Practice (DEEP) (G S ° C) 
Government Services Integration Cluster l 


Chat engagement from Chatbot to Live Chat - Demonstration 


Agent Experience 
Citizen Experience 


MPBSD - ServiceOntario: Enabling Chat Solutions Mid-beta DA-433 GovTechON Ontario Y 


MPBSD 23-071 Record 04 ODS_DFA-AddQuestions- 2022-07-11 Final Page 8 of 37 


1. Understanding Users and Their Needs 1 (GSiC) 


Question - Will there be user testing with live agent scripts? 


= There has been testing in UAT with the DEEP testing team to ensure the 
technology is working 

= Yes - there will be more testing as part of the testing plan to be created by 
DEEP and ServiceOntario for this next phase of this work - plans to test this 
with public before launch will be included 
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1. Understanding Users and Their Needs 2 (GSiC) 


Question - Do you see current SO contact centre agents as users? How will they 
be engaged in this process? 
= Live Chat - We can see agents from the contact centre, both public and hotline, as 
well as in-person CSRs as candidates for agents for live chat, noting there are some 
specific skill sets: 
= Excellent written communication and typing 
= Quick to respond and source information 
= Excellent etiquette and empathy 
= Quick to learn technology and subject 
= Live Chat - ServiceOntario will involve subject matter experts in the build of their 
process and content 
= Chatbot - agents have been at the table for both issues and design for chatbot as 
subject matter experts and continue to contribute 
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2. Establish the Right Team 1 (GSiC) 


Question - Who will be responsible for conducting continued user testing and 

evaluation after Live? 

= ServiceOntario is the business owner and will manage new 
business in partnership with DEEP who support technology 

= Evaluation and measurement is captured in a dashboard 
and ServiceOntario has two measurement teams that can 
monitor this work 

Currently 

= Chatbot team monitors the success of chatbot responses — 
issues are noted and shared 

= ServiceOntario is developing a testing group for work like 
this on their digital team 

= DEEP can provide project support for new queues, 
programs for chat (ie. Health) 


1. Understanding Users and Their Needs 1 (asic) 


. p 
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2. Establish the Right Team 2 (GSiC) 


Question - Do you have UX skills on the team? 


=" DEEP — Has been leading the build 

= The design was done in partnership over the past 3 years with ODS UX designers 

= We have a few people with UX skills, we are gaining maturity, and working with 
ODS and ServiceOntario will have their own people 
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3. Be Consistent 1 (GSiC) 
Question - Please tell us about the content update process for the Ontario.ca pages, the 
chatbot and the Live agent script. Are they in sync and updated at the same time? 


Dave ana Vette Chabot Operatonal Plot 
paste and Content changes Process Flow 


> ReVRe ay 


= = ead 
= see pales = -305 O aeaa = = = = j z 7 [+] 
= = soe be -= $8 am¢ ¢RESSS5 | 


SWAT meetings and an asynchronous Kanban has been 
effective is creating updates for MTO, Drive Test, MGCS, 
ServiceOntario, Web, Social Media, Applications, and 
Contact Centre and ensure updates are synced — new 
content channels can be included. 


12 MPBSD -S ta DA-433 


Approval process has been developed and will continue 
to evolve with ServiceOntario transformation. 
Approved public content can be added to the chatbot. 
New content and requests are submitted to MPBSD 
Comms for approval by MTO policy. 
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(GSiC) 
3. Be Consistent 3 


Question - When are users referred to a live agent? 

= Users can ask to speak to an agent right from the 
start and at any time during the conversation with 
the bot by typing any of the questions that match 
to the “agent” intent in the bot. 

= People are directed to an agent for complex 
questions that a chatbot "knows" it cannot 
answer or if they answer “No” - they did not get 
the answer they are looking for. 

= When a user asks a question that bot can’t 
answer they can try again. On second attempt the E 
user is transferred to an agent. 5 
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(GSiC) 


3. Be Consistent 3 continued 


Question - Are they able to request a live agent from the very beginning of the 
chat? 


14 


Yes, during office hours users are informed up front that live 
support agents are available, and the user can ask to speak to 
an agent right from the start of the conversation. 


If they ask for help, the bot can guide them to ask for a live 
agent. 
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(GSiC) 
3. Be Consistent 3 more 


After hours request for a 
live agent 


= When users begin a conversation with the bot after 
hours it states up front that live support agents are 
not available (and states when they are available). 

= |fa user requests to speak to an agent during the 
conversation, they are reminded that agents are not 
available. 

= They can continue the conversation with the bot if 
they wish to do so. , Sean f 
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3. Be Consistent 4 (GSiC) 


Question - When you conduct the demo, please show us how the front-end 
transitions to the back end to staff view and how it is used and touch points. 


= The demo demonstrated both the citizen and agent view 
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4. Design the Service from Start to Finish — Question 1 (GSiC) 


Question - Can you elaborate on what the division of the responsibility will be 
between your team and the new ServiceOntario team? Now that ServiceOntario 
is in-charge of the chatbot, how will the content/NLU training data be updated? 


ServiceOntario DEEP 

= Daily audit, training, monitoring of the = Participate in weekly issues and 
chatbot content content meetings 

= Issues management, new content = Technical supports and upgrade for EP, 
updates, approvals, and translation IBM Watson, Reports 

=a Monitoring through the reporting in = Project support for adding new lines of 
ICE, IBM Watson, Dashboard business; webpage analysis, gathering 

= Planning for new lines of business and business requirements, technical 
webpage placement design and implementation 

= Page 12 for more detail = Business relationship management 
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4. Design the Service from Start to Finish — Question 2 (GSiC) 


Question - What kind of support will you be providing to the ServiceOntario team 
after the transition over? 


Ongoing Support 

= Like a contact centre or IVR, we continue to provide technical and relationship support 
to the chatbot team 

= If there are issues put in ticket and let the relationship manager know 

= CCOPs with work with our vendor 

= We attend daily chat meetings for flow to Live Chat and Conversational IVR 


Expansion beyond initial scope 
= Any expansion to LOB requires Live Chat to be offered before Chatbot as per Web chat 
guidance - InsideOPS (ontario.ca) 


= There is an established process to update any new business requirements 
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4. Design the Service from Start to Finish — Question 3 (GSiC) 
Question - How will you ensure a smooth onboarding for new team members? 


Agent Training 
= SO training has a robust onboarding and training process for CSRs 
= This includes the business training team for line of business and systems 
= As part of the established training process participants are asks to provide 
anonymous feedback after each training session 
= This is reviewed and used for improvements 


= DEEP has also developed and is piloting training for live chat 
= This will be refined for improvements with ongoing delivery 
= To be shared with ServiceOntario to insert their own operational processes 
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4. Design the Service from Start to Finish — Question 4 (GSiC) 


Question - How will live chat / chat bot answers stay up-to-date based on 
changes in other channels? 


3. Be Consistent 1 (GSiC) 
be 5 Question - Please tell us about the content update process for the Ontario.ca pages, the 
This IS cove red In the chatbot and the Live agent script. Are they in sync and updated at the same time? 
* 6. . - 
Digital Service Standard 3 
question 1 
= .. = ciapa t= m = = 
New content channel to be Zu = = T $eaneeeSSES 
included eo . e 5 = | ae 
== z a 4 
= SWAT me ese cS =| m = 
=" Kanban SWAT meetings and an asynchronous Kanban hasbeen Approval process has been developed and will continue 
effective is creating updates for MTO, Drive Test, MGCS, to evolve with ServiceOntario transformation. 
ServiceOntario, Web, Social Media, Applications, and Approved public content can be added to the chatbot. 
= Approval Process Contact Centre and ensure updates are synced—new New content and requests are submitted to MPBSD 
content channels can be included. Comms for approval by MTO policy. 
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4. Design the Service from Start to Finish — Question 5 (GSiC) 


Question - How will this integrate with the planned video service? 


= Eventually people may access all three using the chat 
window for chatbot, live chat, and video chat 

= This is not currently in scope, as the plan is: 

= Test this video service by screening and inviting 

people 

= Integrate Live Chat with chatbot for D & V 

= Evaluate the pilots, use cases and platforms before 
_ moving forward in partnership with ServiceOntario 
and their business requirements 


Renew a driver's licence 
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6. Test the end-to-end service — Question 1 (GSiC) 


Question - It seems like the team does daily audits, how often are updates made? 
What is the scope of testing involved with each release and how do you make 
sure the changes aren’t impacting the accuracy of the bot for other intents? 


Currently for Chatbot 
= Daily audits are done with the chatbot and result in daily updates. 


= Typically you test other questions previously asked in the changed intent to ensure 
the intent is not impacted. 


= Its not a traditional DEV to UAT to Prod setup — in bot world you go from DEV/UAT to 
Prod (DEV/UAT is a single environment) 


Live Chat 
= Process to be established as part of the project 
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6. Test the end-to-end service — Question 2 (GSiC) 


Question — Can you elaborate more on your regression testing? What’s the scope 
included and are the utterances used to test the intents from real users? 


IBM Watson (for Chatbot) 

= Traditional regression testing is used for introducing a new flow of new content 
when adding to a new webpage, there is a new initiative like VALTAG refunds, or 
when the need arises to split an intent to achieve better coverage. 

= Updates are made to intents and questions regularly in DEV/UAT and tested and 
then moved to production 

= Real user questions as the past 30 days conversations are always available within 
Watson along with records of audits 

Engagement Platform 

= DEEP will perform regression testing along with functional, non-functional, user 
acceptance testing, usability testing - as any updates are made to the application 
as per the regular process 
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7. Make it accessible and inclusive — Question (GSiC) 


Question - Do you know what % of users are on mobile devices? 


= ServiceOntario top web pages 


from June 5 — July 5, 2022 = We are able to determine the 
idac are'belog dead mobile use to the level of the 

webpage 

= Currently the information does 
not roll up to the EP or IBM 
Watson 

= With over 50% of user pressing 
buttons — this action is easier 
on mobile 


Source ServiceOntario web performance dashboard 
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8. Be agile and user-centred — Question (GSiC) 


Question - How will you get feedback from people running the service (e.g. live 
agents) to improve the back-end process? 


= ServiceOntario has a system in place for all staff to participate in improvements 
= All staff are encouraged to use the “Operational Improvement Form” on the staff 
pages in Sharepoint 
= Issues of technical nature would be logged and managed by our CCOps team 
= During projects 
= seeking feedback from the point person embedded in this operation 
= staff focus groups and meetings held 
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9. Use open standards and common platforms — Question 1 (GSiC) 


Question - Where does the data come from for the service? 


Data for live chat is part of the engagement platform. 

= As chat contacts are received, data is written to our common database. This is 
also where we store "feedback." 

When contacts are queued and routed to agents, additional data is stored in the 


"MIMR tables" including information like time in queue, duration of chat with 
agent, etc. 
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9. Use open standards and common platforms — Question 1b (GSiC) 
Engagement Platform 


The engagement platform's reporting tool allows the client and support staff to access out of the 
box contact centre reports and standard chat reports. 


Contact Centre Reports: Standard Chat Reports: 
Queue Contact User Time Report, Chat Feedback Reports: 

Queue Contact Time Report: e Provide information including 
* These reports provide useful feedback scores and "chat 


satisfaction," broken down 
information such as source page, 
agent, or bot. 

Chat Feedback Comment Reports: 


information such as the "grade of 
service," the percentage of chats 
answered within 45 seconds. 


Queued Contacts Summary Report: * Show comments left by users along 
* Shows percent of users that left with their feedback rating. 

the chat before speaking with an 

agent. 
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9. Use open standards and common platforms — Question 2 (GSiC) 


Question - What is the process for getting information into the service? 


= Live Chat "Canned Responses" 


28 


The web chat framework provides a standard set of responses that can be 
used to provide a consistent user experience across OPS chat solutions, and 
these are loaded to the system during the project along with any 

other specific responses that have been developed/vetted during the project. 
Canned responses can be used by the agents to respond quickly and 
consistently. 

Over time responses can be deleted, modified, added as the need arises. 
Once in operational mode, ServiceOntario will submit a SODO request to 
update canned responses, and the change by DEEP support staff involves 
updating "iceBar configuration." 

Once the change is complete it can be accessed by all live agent staff. 
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11. Support those who need it — Question 1 & 2 


Question - How will live agents be trained? 
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(GSiC) 


Question - Are there any training feedback loops available to improve responses? 


4. Design the Service from Start to Finish — Question 3 (GSiC) 
Question - How will you ensure a smooth onboarding for new team members? 


Agent Training 
= SO training has robust onboarding and training process for CSRs 
= This includes the business training team for line of business and systems 
= As part of the established training process participants are asks to provide 
anonymous feedback after each training session 
= This is reviewed and used for improvements 


= DEEP has also developed and is piloting training for live chat 
a This will be refined for improvements with ongoing delivery 
= To be shared the ServiceOntario to insert their own operational processes 
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12. Measure performance — Question 1 (GSiC) 
Question - With live chat coming, is the chatbot worth maintaining? 


In short YES - the chatbot: 
= Answers questions 24/7, is a way to quick self-serve, and is giving the right answer 91% of 
the time 
= Has inspired future developments both operationally, and for other digital products 
= This is a voice of the customer tool for issues and content needs assisting communication 
across all service channels 
= Supported the sticker refund program identifying questions by the public and issues with 
information: 
= How to apply? 
= Where is my cheque? — my neighbour has theirs 
= Why is my cheque so big/small/wrong 
= [have lost or damaged my cheque 
a Inconsistency on the webpages about renewals in-person and online 


= Live Chat is not a 24/7 operation 
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12. Measure performance — Question 2 (GSiC) 


Question - What will it cost to maintain? What’s the ROI? 


= It costs 1.5 — 2.0 FTE to maintain the chatbot for the current volume 
= Uses a consumption based model based on number of chats — dependant on the 
platform 
= Since users are anonymous you cannot follow them through the full service to the 
phone or in-person, therefore the ROI is based on assumptions 
= Calculations with assumptions support an increased capacity at the Contact 
Center that is equal to the effort that it takes to maintain it. 
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12. Measure performance — Question 3 (GSiC) 


Question - Have you set KPIs and targets for the live chat feature? 


=" The chatbot dashboard has KPIs for satisfaction, effectiveness, adoption — submitted for 
DFA 
= Live chat - the objective is to answer questions fully with no calls to the dedicated 
chatbot line - calls should go close to zero 
=" ServiceOntario has sophisticated measurements groups that will input to this 
= Benchmark KPIs 
= Satisfaction — from standard Chat Feedback reports 
= calls from the chatbot 


Benchmark chat key performance indicators can be used by all contact centers to measure the success of tho implementation 


H H Grade of Service (GOS) of 80% or greater for chat queues - Target Average Speed of Answer for a chat contact is 45 
= chat resolution time seconde. Standard reports show the contact genre adherence 1o ihe target wih Grade of Service (GOS). To nd grade of 
A H Qu Ce User Time Report 
= first response time Quove Contact Time Report 
Pe ese «Ghat Satiataction Rate of 80% or greater — Chat Feedback Report racks the number of web users that submited a feedback 
Average wait time in queue «Chat queue Abandonment Rate ofS or less Abandoned chais are web users that were waling In queue bu closed the 


= queue (Le. out of total contacts offered to the queue). The contact centre can look at this statistic to determine adherence to 
Webpage visits to chat and live chat ge ea al ele ee 
Te aea beraiena 


= Missed chats Quoued Contacts Summary Report 


ta DA-433 C N Ontario Y 


MPBSD 23-071 Record 04 ODS_DFA-AddQuestions- 2022-07-11 Final Page 33 of 37 


12. Measure performance — Question 4 (GSiC) 


Question - Please share any details on how you are planning to measure success 
for the new feature. 


During the project - Phase II — Live Chat (to be confirmed by the Project Team) 
= Daily review of Chats for quality and accuracy 
= Look for 


= Increase in satisfaction 

Less calls from the chatbot 
Decreasing chat resolution time 
Quicker first response time 
Impact on other channels 
Issues and tickets 


= Typically recommend using the feedback to understand the voice of the customer 
= ServiceOntario has sophisticated measurements groups that will input to this 
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12. Measure performance — Question 4b (GSiC) 


Question - Please share any details on how you are planning to measure success 

for the new feature. 

The standard approach sees the project team collaborating to document in the solution design how success 
will be measured, including metrics, measures, baseline measurements, target approach, and reporting cycle. 
This work has been initiated and will be reviewed as ServiceOntario re-engages in the project: 


A draft excerpt: 

The project team has defined key performance indicators that will measure the success of the chat deployment. These KPIs allow the team 
to measure how well the objectives are being met (i.e., looking back to the business analysis, is the pain point addressed or the opportunity 
achieved?): 

Problem statement: Improve the experience in the current D & V chatbot for people who are looking to quickly self-serve for information, 
or to get reassurance on what they find online about Driver and Vehicle services. 

To measure the improvement of the customer experience with the implementation of the D & V Live Chat the team will take three months 
of metrics (% Satisfied from the Chat Feedback Report, and Early Abandons) to form a baseline in which to measure the success of the 
project. 

The goal will be to the measure metrics % Satisfied increased and decrease in the Early Abandon rate. This would an indicator that the 
project has Improved the Customer Experience from that of the current D & V Chatbot. 

The KPI belongs here (e.g. % Satisfied from the Chat Feedback Report and “Early Abandons — see the spreadsheet) and would be good to 
have details on how we are getting the baseline. 
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Digital Enterprise Engagement Practice (DEEP) (G S ° C) 
Government Services Integration Cluster l 


Webpage Deployment: 
5 D&V webpages in both French and English were selected for pilot 
2 New pages for Licence plate refund initiative 
Jan 25, 2021 -> Register Vehicle Permit Licence Plate and Sticker (https://www.ontario.ca/page/register-vehicle-permit-licence- 
plate-and-sticker) 
Feb 25, 2021 -> Buy or Sell Used Vehicle Ontario (https://www.ontario.ca/page/buy-or-sell-used-vehicle-Ontario 
April 8, 2021 -> 
Renew Driver’s Licence (https://www.ontario.ca/page/renew-drivers-licence) 
+  Driver’s Licence (https://www.ontario.ca/page/drivers-licence 
Payment Help: Driver & vehicle online services (https://www.ontario.ca/page/payment-help-driver-and-vehicle-online- 
services) 
April 1, 2022 -> 
Renew Licence Plate Sticker (https://www.ontario.ca/page/renew-licence-plate-sticker ) 
Get a refund for your licence plate sticker (https: 
or-drivers-licence ) 


age/get-refund-or-credit-your-licence- 
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Chatbot New Webpages (GSiC) 
New pages added on April 1, 2022 


» Renew Licence Plate Sticker (https://www.ontario.ca/page/renew-licence-plate-sticker 


Get a refund for your licence plate sticker (https://www.ontario.ca/page/get-refund-or-credit-your-licence-plate-sticker-or-drivers-licence ) 


e Chats have gone from over 200 to 800 on some days — Cheques are being received for sticker refunds 


e This is spike of interest with a focus on the cheques - social media is potentially driving the people to 
question their cheque amounts 


Top of questions ‘No’s” because? 


® How to apply? 
® Where is my cheque? - my neighbour has theirs 


® Why is my cheque so big/small/wrong * No public statements telling people how they are 


® | have lost or damaged my cheque being distributed, some believe it is the Birthday, 
* Inconsistency on the webpages about renewals in- others the alphabet 


person and online * Social media and uncertainty fuel the questions 
about the amount not being correct 


* Need to speak with an agent - Personal answers 
not in scope or chatbot ability 


eta DA-433 


Ontario WY 


